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(54) Super resolution apparatus and methods for electro-optical systems 



(57) There is disclosed in an optical system having 
a predetermined Numerical aperture which provides a 
corresponding level of spatial resolution and having 
detector means and processor means in which image 
data is obtained comprising noisy blurred scene data 
containing an object to be reconstructed, and noisy 
blurred background data of the same scene, an 
improved method for increasing the spatial resolution of 
the imaging data produced by the diffraction limited opti- 
cal system. The improvement comprises filtering the 
noisy blurred background data of the same scene to 
obtain noise suppressed data; applying estimations of 
point spread functions associated with the noise sup- 
pressed data and optical system to estimates of the 
noise suppressed data to obtain a reconstructed back- 
ground image (Wx)); and low pass filtering the noisy 
blurred scene data containing the object to be recon- 
structed (D1) and using the reconstructed background 
image (l r (x)) to eliminate the background data from the 
image data to obtain a reconstructed image of an object 
with increased spatial resolution. 
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Description 

RELATED APPLICATIONS 

[0001 ] The present application is related to copending 
commonly assigned U.S. patent application, serial 
number 08/763,610. filed on December 11, 1996, enti- 
tled "Apparatus and Method For Providing Optical Sen- 
sors With Super Resolution", incorporated herein by 
reference. 

FIELD OF THE INVENTION 

[0002] The present invention relates generally to opti: 
cal devices, and more particularly to an optical sensor 
utilizing background reconstruction image processing 
techniques in order to provide a much higher level of 
resolution of a localized object within a 5 background 
scene. 

BACKGROUND OF THE INVENTION 

[0003] Optical sensors are devices which for decades 
have been used to detect and record optical images. 
Various types of optical sensors have been developed 
which work in the Ultra Violet Bands. Infra Red Bands 
as well as in the Visible Bands of operation. Examples 
of such devices include Weather Sensors. Terrain Map- 
ping Sensors, Surveillance Sensors, Medical Probes, 
Telescopes and Television Cameras. 
[0004] An optical sensor typically includes an optical 
system and one or more detectors. The optical system 
portion is made up of various combinations of lenses, 
mirrors and filters used to focus light onto a focal plane 
located at the image plane of the optical system. The 
detectors which make at the image plane are used to 
convert the light received from the optical system into 
electrical signals. Some types of optical sensors use 
film rather than detectors to record the images. In this 
case, the grain size of the film is analogous to the detec- 
tors described above. 

[0005] An important performance characteristic of 
optical sensors is their "spatial resolution" which is the 
size or the smallest object that can be resolved in the 
image or, . equivalent^, the ability to differentiate 
between closely spaced objects. If the optical system is 
free from optical aberrations (which means being "well 
corrected") the spatial resolution is ultimately limited by 
either diffraction effects or the size of the detector. 
[0006] Diffraction is a well known characteristic of light 
which describes how light passes through an aperture 
of an optical system. Diffraction causes the light passing 
through an aperture to spread out so that the point light 
sources of a scene end up as a pattern of light (known 
as a diffraction pattern) diffused across the image. For a 
well corrected, unobscured optical system known as a 
diffraction limited system, the diffraction pattern 
includes a very bright central spot, surrounded by 



somewhat fainter bright and dark rings which gradually 
fade away as the distance from the central spot 
increases. 

[0007] An optical sensor that is designed to be diff rac- 

5 tion limited usually has a very well corrected optical sys- 
tem and detectors sized so that the central spot of the 
diffraction pattern just fits within the active area of the 
detector. With conventional sensors, making the detec- 
tors smaller does not improve resolution and considera- 

io biy increases the cost due to the expense of the extra 
detectors and the associated electronics. 
[0008] The size of the aperture used in the optical sys- 
tem determines the amount of resolution lost to diffrac- 
tion effects In applications such as camera lenses and 

is telescope objectives, the aperture size is normally 
expressed as an f -number which is the ratio of the effec- 
tive focal length to the size of the clear aperture. In 
applications such as microscope objectives, the aper- 
ture size is normally expressed as a Numerical aperture 

20 (NA) which is the index of refraction times the sine of the 
half angle of the cone of illumination. For a given local 
length, a high f -number corresponds to a smaller aper- 
ture, while a higher Numerical aperture corresponds to 
a larger aperture 

25 [0009] A basic limitation of conventional optical sen- 
sors is the aperture size required for a given level of res- 
olution. Higher resolution images require larger 
apertures. In many situations the use of such a system 
is very costly. This is because using a larger aperture 

30 requires a significantly larger optical system. 

[0010] The cost for larger systems which have aper- 
tures with diameters greater than one foot is typically 
proportional to the diameter of the aperture raised to a 
power of "x M . The variable "x w usually ranges from 2.1 to 

35 2.9 depending on a number of other particulars associ- 
ated with the sensor such as its wave band, field of 
regard, and field of view. 

[001 1 ] The size of the optical sensor is particularly rel- 
evant for systems that fly on some type of platform, 

40 either in space or in the air. Under such conditions the 
sensor must be lightweight, strong, and capable of sur- 
viving the rigors of the flight environment. Thus the cost 
of going to a larger optical system can be in the hun- 
dreds of millions of dollars for some of the larger and 

45 more sophisticated sensors. Practical considerations, 
such as the amount of weight the host rocket, plane, 
balloon, or other vehicle can accommodate, or the 
amount of space available, may also limit the size of the 
sensor. These practical considerations can prevent a 
. so larger system from being implemented no matter how 
large the budget. 

[0012] A number of optical imaging techniques have 
been developed to increase spatial resolution. One 
such technique is known as sub-pixel resolution. In sub- 
55 pixel resolution the optical system is limited in spatial 
resolution not by diffraction but by the size of the detec- 
tors or pixels. In this case, the diffraction pattern of the 
aperture is much smaller than the detectors, so the 
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detectors do not record all the resolution inherent in the 
optical system's image. Sub-pixel resolution attempts to 
reconstruct an image that includes the higher resolution 
not recorded by the detectors. This technique does not 
require hardware or system operation changes in order 5 
to work. Examples of sub-pixel resolution techniques 
are disclosed in an. article in ADVANCED SIGNAL 
PROCESSING ALGORITHMS, ARCHITECTURES 
AND IMPLEMENTATIONS II, by J.B. Abbiss et aL, The 
International Society For Optical Engineering, Volume ^10 
1566. P. 363 (1991). 

[0013] Another example is the use of "thinned aper- 
ture" systems where for example, a widely-spaced pat- 
tern or small holes is used as a substitute for the 
complete aperture. However, even "thinned apertures" 15 
are limited in resolution by diffraction theory and by the 
outer diameter of the widely-spaced pattern or small 
holes. Note that current electro-optical systems are 
sometimes designed so that the size of their detector 
matches the size of the diffraction blur of their optics. 20 
[0014] Other examples of optical imaging techniques 
are disclosed in an article entitled SUPER RESOLU- 
TION ALGORITHMS FOR A MODIFIED HOPFIELD 
NEURAL NETWORK, by J.B. Abbiss et al. a IEEE Trans- 
actions On Signal Processing, Vol. 39, No. 7, July 1991 2s 
and in a paper entitled FAST REGULARIZED DECON- 
VOLUTION IN OPTICS AND RADARS, by J.B. Abbiss 
et al.. presented at the 3rd IMA Conference on Mathe- 
matics in Signal Processing. These techniques use lin- 
ear algebra and matrix techniques to restore signals or 30 
images from a limited discrete data set. 
[0015] The previously described techniques have a 
number of drawbacks with regard to optical sensors. 
First, only one of these techniques is directed toward a 
diffraction limited device. In addition, these techniques 35 
often produce systems of equations which cannot be 
solved due to the practical constraints on computing 
power. Furthermore, none of the previously described 
techniques specify either the types of detectors or other 
system parameters which are used along with these 40 
techniques. 

[0016] It is, therefore, an object of the present inven- 
tion to provide an apparatus and method for increasing 
the resolution of an optical sensor without using a sub- 
stantially larger aperture. 

SUMMARY OF THE INVENTION 

[0017] The invention provides an apparatus and 
method for improving the spatial resolution of an object so 
using a background reconstruction approach wherein a 
localized object containing high spatial frequencies is 
assumed to exist inside a background scene containing 
primarily low and/or very high spatial frequencies com- 
pared to the spatial frequencies of the localized object, ss 
The imaging system cannot pass these high spatial fre- 
quencies (neither the high frequencies of the object, nor 
the very high frequencies of the background) . The 



background image's low spatial frequencies are used to 
reconstruct the background scene in which the localized 
object is situated. Using this reconstructed background 
and the space limited nature of the localized object (i.e. 
it is only present in part of the scene, not the entire 
scene), the high spatial frequencies that did not pass 
through an optical system are restored, reconstructing a 
detailed image of the localized object. 
[0018] An improvement comprises filtering the noisy 
blurred background data of the same scene to obtain 
noise suppressed data; applying estimations of point 
spread functions associated with the noise suppressed 
data and optical system to estimates of the noise sup- 
pressed data to obtain a reconstructed background 
image (l r (X)); and low pass filtering the noisy blurred 
scene data containing the object to be reconstructed 
(D1, and using the reconstructed background image 
(l r (x)) to eliminate the background data from the image 
data to obtain a reconstructed image of an object with 
increased spatial resolution. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[001 9] The above objects, further features and advan- 
tages of the present invention are described in detail 
below in conjunction with the drawings, of which: 

FIGURE 1 is a general block diagram of the optical 
sensor according to the present invention; 
FIGURE 2 is a diagram of a linear detector array tai- 
lored to the present invention; 
FIGURE 3 is a diagram of a matrix detector array 
tailored to the present invention; 
FIGURE 4 is a diagram illustrating the size of an 
individual detector tailored to the present invention; 
FIGURE 5 is a diagram illustrating how the detector 
grid is sized with respect to the central diffraction 
lobe; 

FIGURE 6 is a diagram of a multi-linear detector 

array tailored to the present invention; 

FIGURE 7 is a diagram of another version of a 

multi-linear detector array tailored to the present 

invention; 

FIGURE 8 is diagram illustrating the operation of a 
beam splitter as part of an optical system tailored to 
the present invention; 

FIGURE 9A-B represents an image scene and a 
flow diagram of the non-linear background recon- 
struction technique of the image scene according to 
the present invention; 

FIGURE 10 is a diagram illustrating the Richard- 
son-Lucy background reconstruction portion of the 
non-linear image processing technique according 
to the present invention; 

FIGURE 11 is a diagram illustrating the Object 
Reconstruction and background subtraction portion 
of the non-linear image processing according to the 
present invention; 
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FIGURE 12 is a schematic illustrating the Fourier 
Transform characteristic of binotf. 
FIGURE 1 3 is a schematic of the relationship of bin- 
map to the object of interest within a window of a 
particular scene according to the present invention; 
and 

FIGURES 14A-D show the effects of super-resolu- 
tion of simulated bar targets using the non-linear 
reconstruction technique according to the present 
invention; 

FIGURES 15A-D show the Fourier analysis of sim- 
ulated bar targets associated with FIGs 14A-D; 
FIGURES 16A-C show simulated images associ- 
ated with a thinned aperture optical system config- 
uration using non-linear method. 
FIGURES 17A-C show the super-resolution of an 
image of a figure taken from a CCD camera and 
reconstructed using the non-linear method accord- 
ing to the invention; 

FIGURES 17D-F show the Fourier image analysis 
of the image of FIGs 1 7A-C ; 
FIGURES 18A-B show the Fourier analysis of 
superimposed truth, blurred, and reconstructed 
images as function of frequency for SNR values of 
50 and 100. respectively; 

FIGURES 19a.b are flow diagrams of the linear 
algebra background reconstruction technique 
according to the present invention. 

DETAILED DESCRIPTION OF THE INVENTION 

[0020] The present invention is directed to an appara- 
tus and method of superresolution of an image for 
achieving resolutions beyond the diffraction limit. This is 
accomplished by background reconstruction, wherein a 
localized object containing high spatial frequencies is 
assumed to exist inside a background scene containing 
primarily low and/or very high spatial frequencies com- 
pared to the spatial frequencies of the localized object. 
The point spread function (PSF) of an optical system 
causes the background data to blend with and flow over 
objects of interest, thereby contaminating the bounda- 
ries of these objects and making it exceedingly difficult 
to distinguish these objects from the background data. 
However, if one knows or can derive the background, 
then the object may be disentangled from the back- 
ground such that there will exist uncontaminated bound- 
aries between the object and the background scene. 
Another way of describing the effect of a PSF is to note 
that the spatial frequencies associated with the object of 
interest which lie beyond the optical system cutoff fre- 
quency are lost. In the background reconstruction 
approach, a localized object such as a tractor having 
high spatial frequencies is super-resolved inside a back- 
ground scene containing primarily low and/or ultra-high 
spatial frequencies such as a cornfield. Typically, the 
imaging system cannot pass either the high spatial fre- 
quencies of the object or the ultra-high spatial frequen- 



cies of the background. This invention uses the 
background images' low spatial frequencies to recon- 
struct the background scene in which the localized 
object is situated. This reconstructed background and 

5 the space limited nature of the localized object (that is, 
the object is present in only part of the scene rather than 
the entire scene) can be used to restore the high spatial 
frequencies that did not pass through the optical sys- 
tem, thereby reconstructing a detailed image of the 

10 localized object. 

[0021 ] In accordance with alternative embodiments of 
the present invention, both linear and nonlinear meth- 
ods for reconstructioning localized objects and back- 
grounds to produce super-resolved images are 

is described. 

[0022] Referring to FIGURE 1 , there is shown a gen- 
eral block diagram of an optical sensor accommodating 
the present invention. The sensor (10) as in conven- 
tional devices includes an optical system (12) and 

20 detectors (14). The optical system (12) which includes 
various combinations of lenses, mirrors and filters 
depending on the type of application is used to focus 
light onto a focal plane where the detectors (14) are 
located. The optical system (12) also includes a prede- 

25 termined aperture size corresponding to a particular 
Numeral aperture (NA) which, in conventional diffrac- 
tion-limited devices, limits the amount of spatial resolu- 
tion that is attainable. This is. as previously described, 
due to diffraction blurring effects. 

30 [0023] The optical system (12) can be described by an 
optical transfer function (OTF) which represents the 
complete image forming system, and which can be 
used to characterize that system. 
[0024] The detectors (14) convert the light received 

35 from the optical system (12) into the electrical signals 
which become the data used to generate images. In 
conventional sensors the detectors are configured in a 
linear array for Scanning systems or in a matrix array for 
Staring systems. In Scanning systems, the detector lin- 

40 ear array is swept in a direction perpendicular to the 
length of the array generating data one scan line at a 
time with each line corresponding to one line of the 
image. In Staring systems the matrix array is not moved 
and generates all of the imaging data simultaneously. 

45 Thus each detector of the matrix array corresponds to 
one pixel of the image. It is intended that the detectors 
(14) of the present invention will be configured as a lin- 
ear array or a matrix array depending on the type of sys- 
tem being used. 

so [0025] The detectors (14) take many different forms 
depending on the wavelength of light used by the 
present invention. For example, in the ultraviolet and X- 
Ray ranges such detectors as semitransparent photo- 
cathodes and opaque photocathodes can be used. In 

55 the visible range such detectors as vacuum phototubes, 
photomultipliers, photoconductors, and photodiodes 
can be used. In the infrared range, such detectors as 
photoconductors, photodiodes. pyroelectric, photon 
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drag and golay cell devices can be used. 
[0026] In the present invention various elements of the 
sensor (10) must be optimized to be used with a partic- 
ular image-processing technique. The type of optimiza- 
tion depends on the image-processing technique. As 
will be described in detail later, the present invention 
includes two alternative image-processing techniques. 
Each of these two super-resolution methods may be 
used with the sensor configurations as described below. 
[0027] In case one, the sensor (10) must include 
detectors (14) that have an "instantaneous field of view" 
that is equal to or less than the desired level of spatial 
resolution. If, for example, the required resolution is one 
meter or less then the "instantaneous field of view" of 
the detectors must be one meter or less (even though 
the central lobe of the diffraction pattern is much larger). 
This makes the pixel size of the image produced by the 
sensor (10) smaller than the central diffraction lobe. 
(Note that, such a configuration adds additional cost to 
the sensors. However, for large systems the increase in 
cost is less than the cost of a larger aperture.) 
[0028] The sensor ( 1 0) can obey this rule in one of two 
ways. One way is to use more smaller-size detectors 
(14). In conventional sensors the number of detectors 
used varies anywhere from one to millions depending 
on the application. 

[0029] In one embodiment of the present invention at 
least five times more detectors (14) than normal are 
required to achieve the desired resolution. A diagram of 
a linear detector array to be used with the present inven- 
tion is shown in FIGURE 2, while a diagram of a matrix 
detector array to be used with the present invention is 
shown in FIGURE 3. The number of detectors (14) 
included in these arrays (28), (30) depends on the appli- 
cation. However, as previously pointed out to achieve 
the higher resolution these arrays (28), (38) will include 
at least five times more detectors (14) than conventional 
sensors for a given application. 
[0030] In conventional sensors, the size of the individ- 
ual detector is never smaller than the size of the central 
diffraction on lobe . This is because utilizing smaller 
sensors serves no purpose since the resolution is lim- 
ited by the optical aperture. In the present invention, the 
size of the individual detectors (14) must be smaller 
than the size of the central diffraction lobe (18), as 
shown in FIGURE 4. 

[0031] Another way of configuring the sensor (10) 
according to Case one is again to use a larger number 
of detectors (14), but instead of using smaller detectors 
configure the optical system (12) so that more than one 
detector is spread across the central diffraction lobe. 
This allows conventional size detectors (14) to be used. 
The number of detectors (14) used again must be five 
times or more than required in conventional sensors. In 
order to configure the optical system (12) as described 
above the back focal length must be adjusted so that 
five or more detectors (14) spread across the central dif- 
fraction lobe (18), as shown in FIGURE 5. 



[0032] In Scanning systems, it is difficult to generate 
multiple image data by viewing the object at different 
times. This is because the optical system assumes that 
the object remains stationary while being scanned. The 

s solution is to use detectors (14) configured in a multi-lin- 
ear array as shown in FIGURE 6. The multi-linear array 
(20) includes a number of individual linear arrays (22) 
shifted parallel to each other by a distance (d) which is 
a fraction of a pixel. When the array (20) is swept each 

10 linear array (22) generates imaging data corresponding 
to one scan line of each of the images. Thus each linear 
array (22) creates one of the images being produced. In 
the preferred configuration, the array (20) includes ten 
or more individual linear arrays 22 which are capable of 

/5 producing ten or more different images. 

[0033] Now two more examples of this technique are 
discussed. These examples are labeled Case three and 
Case four. 

[0034] In Case three, the sensor (10) is configured to 

20 take images of objects in multiple color bands. In Scan- 
ning systems, this is accomplished by utilizing a multi- 
linear array, as shown in FIGURE 7. The multi-linear 
array (24) also includes a number of individual linear 
arrays (22) arranged in a parallel configuration shifted 

25 parallel to each other by a distance (d) which is a frac- 
tion of a pixel. A color filter (26) is disposed over each of 
the linear arrays (22). The color filters (26) are config- 
ured to pass only a particular portion of the color spec- 
trum which may include multiple wavelengths of visible 

30 light. When the array (24) is swept, each linear array 
(22) produces images of the same object in different 
color bands. The filters (26) are fabricated by depositing 
optical coatings on transparent substrates, which are 
then placed over each of the linear arrays (22). This 

35 process is well known. 

[0035] In Staring systems, the multiple color-band 
data is created by incorporating a beam splitter in the 
optical system (12) and using multiple detector arrays. 
Such a configuration is illustrated in FIGURE 8. The 

40 beam splitter (28) splits the incoming light (32) into mul- 
tiple beams (34); Figure 8 shows the basic idea using a 
two-beam system. Due to the operation of the beam 
splitter (28) each light beam (34) includes a different 
part of the color spectrum which may include one or 

45 more different bands of visible or infrared light. Each 
light beam is directed to one of the detector arrays (30) 
producing images of the same object in different color 
bands. 

[0036] In Case four, the sensor (10) is a combination 
so of the three previously descrtoed cases. This is accom- 
plished by combining the principles discussed above 
with regard to Cases one, two or three. In all three 
cases the sensor must be desired to have a signal to 
noise ratio which is high as possible. This is done either 
55 by increasing the integration time of the detectors (14) 
or by slowing down the scan speed as much as possible 
for scanning systems. For Case two, the system's 
design, or its operation mode, or both, are changed in 
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order to take the required multiple images in a known 
pattern displaced by a known distance that is not a mul- 
tiple of a pixel, but rather is a multiple of a pixel plus a 
known fraction of a pixel. 

[0037] Referring back to FIGURE 1, coupled to the 
detectors (14) is a processor (16) which processes the 
image data to achieve the higher resolution. This done 
by recovering "lost" information from the image data. 
Even though the diffraction blur destroys the required 
spatial resolution, some of the "lost" spatial information 
still exists spread across the focal plane. The small-size 
detectors described above are used to sample at a 5 to 
10 times higher rate than is customary in these sorts of 
optical systems in conjunction with processing enabling 
much of this "lost" information to be recovered and thus 
restoring the image to a higher level of resolution than 
classical diffraction theory would allow. 
[0038] The processor (16) uses one of two image 
processing techniques: a Non-linear Reconstruction 
method using a modified Richardson -Lucy Enhance- 
ment technique, and a background reconstruction 
approach using a linear algebra technique. 
[0039] One reasonable extension of the previously 
described imaging techniques is to use phase retrieval 
or wave front phase information to reconstruct the 
image and thus achieve higher resolution. Another rea- 
sonable extension of the previously described tech- 
nique is to use prior knowledge of the background 
scene to help resolve objects that have recently moved 
into the scene. The processor (16) in addition to using 
one of the above described primary data processing 
techniques, also uses other techniques to process fur- 
ther the imaging data. This further processing is accom- 
plished by standard image enhancement techniques 
which can be used to improve the reconstructed image. 
Such techniques include, but are not limited to. edge 
sharpening, contrast stretching or other contrast 
enhancement techniques. 

[0040] The Non-linear Background Reconstruction 
method using a modified Richardson-Lucy Enhance- 
ment Technique is described as follows. In Figure 9A, 
there is shown a scene (S) comprising a localized object 
(O) such as a tractor within a noisy blurred background 
(B). In Figure 9B. input data D2 (Block 20). representing 
the noisy blurred background data, is input into module 
30 to remove the noise from D2 using a modified ver- 
sion of the method of sieves. Note that the input data D1 
and D2 indicated in block 20 has been sampled at the 
Nyquist rate (5 times the customary image sampling 
rate) and preferably at twice the Nyquist rate (ten times 
the customary sampling rate) to obtain robust input 
data. 

[0041] The modified method of sieves removes noise 
by averaging adjacent pixels of the noisy blurred back- 
ground data D2 of the same scene together, using two 
and three pixel wide point spread functions. Array ho in 
equation (9a) of module 30 represents the optical sys- 
tem PSF. which is the Fourier transform of the OTF 



input. 

[0042] The output of module 30 thus provides sepa- 
rate pictures - the optical system image l(x) and the 
modified background data images D3(x) and D4(x). As 

5 shown in block 40, new point spread functions h 3T and 
h 4T have been constructed to account for the combined 
effect of the method of sieves and the optical system. 
These new point spread functions use both the two and 
three pixel wide point spread functions, h 3 and h 4 , pre- 

10 viously identified, as well as the optical system PSF h 0 , 
as shown in equation 9(d) and (e), to arrive at the new 
PSFs. 

[0043] As shown in block 50, the Richardson-Lucy 
method is then used to reconstruct the background 

75 scene data. D2, with the reconstructed data defined to 
be l r (x). Figure 10 shows an exploded view of the 
processing steps for reconstructing the background 
scene to obtain the reconstructed background l r (x). 
[0044] In Figure 10 the noise suppressed data D2 

20 from block 20 of Figure 9B, is used as the first estimate 
of the true background scene, l n (x) (Module 100). As 
shown in module 110, this estimate of the true back- 
ground scene is then blurred using the combined 
method of sieves and optical system point spread func- 

25 tions to obtain two picture representations 13(x) and 
14(x). where 13(x) is given by equation 10 (a) and 14 (x) 
is given by equation 10 (b). Two new arrays, as shown in 
module 120. are then created by dividing pixel by pixel 
the noisy scene data, D3(x) and D4(x), by the blurred 

30 estimate of the true background scene, I3(x) and I4(x), 
as shown in equations 10(c) and (d) respectively. The 
new arrays T3(x) and T4(x) are then correlated with the 
combined method of sieves and optical system PSF's 
and the result is multiplied pixel by pixel with the current 

35 estimate of the true background scene. In this manner a 
new estimate of the true background scene, Z(x), is 
obtained as shown in module 130 and by equations 10 
(e)-(g). 

[0045] The processor then determines whether a pre- 

40 determined number of iterations have been performed, 
as shown in block 140. If the predetermined number has 
not been performed, the current estimate of the true 
background is replaced by the new estimate of the true 
background (Z(x)) (Block 150) and the processing 

45 sequence or modules 110-140 are repeated. If, how- 
ever, the predetermined number has been reached, 
then the latest estimate of the true background scene is 
taken to be the reconstructed background scenes; that 
is, l r (x) is taken to equal Z(x) as shown in block 160. 

so Note that for the optical systems described above the 
predetermined number of iterations is preferably 
between one thousand and two thousand. 
[0046] In Figure 9B the reconstructed background l r (x) 
(equal to Z(x)) is then input to module 60. where the 

55 background subtraction steps are performed to super- 
resolve the object within the noisy blurred scene data 
(D1). The reconstructed object, G(x), which has been 
super-resolved, is then output as shown in module 70. 
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[0047] The background subtraction method for super- 
resolving the object of interest in D1 is detailed in Figure 
11. In Figure 11 the noisy blurred scene data D1 con- 
taining the object to be reconstructed is used as input, 
with a low-pass filter (block 600) applied to remove high 
spatial frequency noise from the D1 data as shown by 
equation 11(a). In equation 11(a) the transfer function 
h b (x) represents the Fourier transform of the binotf 
array The binotf array specifies the non-zero spatial fre- 
quency of the otf located in the fburier plane. Figure 12 
shows the fburier transform of an image depicting binotf 
to ensure no higher frequencies in the spectrum exist 
[0048] As shown in Figure 12, the values of binotf are 
1 up to the optical system cutoff value f c and 0 beyond 
that cutoff. The low pass filtered data D f (x) in Figure 1 1 
is then multiplied pixel by pixel with the bin man array 
(binmap (x)) to separate out a first estimate of the 
reconstructed object from the filtered D1 data, as shown 
in equation 11(b) in module 610. The binmap array 
specifies the region in the scene containing the object to 
be super-resolved. The binmap has array elements 
equal to 1 where the object of interest is located and 
array elements equal to 0 everywhere else. Figure 13 
shows a pictorial representation of the binmap array, 
where the binman window (W) holds a region containing 
an object of interest (Rl) consisting of pixels equal to 
one in the region containing the object and pixels equal 
to zero everywhere else (RO). 
[0049] The next step in Figure 1 1 is to replace the 
reconstructed background scene pixels, l r (x), by the 
estimated reconstructed object pixels, D 0 (x), at object 
positions specified by the binmap (40), as shown in 
module 620. Equation 11© provides the mathematical 
formula for this replacement, creating a reconstructed 
object array S(x). S(x) is then convolved with the optical 
system PSF (h 0 ) to blur the combination of the recon- 
structed background and the estimated reconstructed 
object as shown in equation 1 1 (d) of module 630. A new 
array N(X), is then created in block 640 by dividing, on 
a pixel by pixel basis, the filtered D1 scene array (D f ) by 
the blurred combination of the reconstructed back- 
ground and the estimated reconstructed object (l s ) as 
shown in equation 1 1(e) of module 640. The new array, 
N(x), is then correlated with the optical system PSF (ho) 
and multiplied, for each pixel specified by binmap, by 
the current estimate of the reconstructed object. Equa- 
tion 1 1(f) of module 650 is then used to determine Fix), 
the new estimate of the reconstructed object. 
[0050] After K(x) has been calculated a check is 
made, as shown in module 660, to determine whether 
the specified number of iterations have been completed. 
If more iterations are needed the current estimate of the 
reconstructed object is replaced by the new estimate as 
shown by equation 1 1 (g) of module 670. Steps 620 - 
660 are repeated until the specified number of iterations 
have been accomplished. When this happens the latest 
estimate of the reconstructed objects is taken to be the 
reconstructed object; that is G(x) is set equal to K(x), as 



shown in module 680 (equation 1 1H). 
[0051] In Figure 9B the reconstructed object G(x) is 
the output of module 70. This G(x) is in fact the desired 
super-resolved object. Note that the above description 

s of the super-resolution method as shown in Figure 1 1 is 
set up to handle non-thinned apertures. For thinned 
aperture systems, step 630 of Figure 1 1 (in which the 
new scene l 8 (x) is blurred again using the optical sys-. 
tern's PSF) may be excluded. 

w [0052] Three examples of applying the background 
reconstruction approach using the above-described 
non-linear technique to obtain surer-resolution are illus- 
trated in Figures 14-16. Figures 14A - D represent sim- 
ulated bar target charts where Figure 14A represents 

is the truth scene illustrated by a series of alternating dark 
and light bands within a background. Figure 14B repre- 
sents the blurred image of the truth scene (through a 
small aperture), and Figures 14C and 14D represent 
the reconstructed images using the non-linear back- 

20 ground reconstruction method previously described. 
Figures 15A - D are associated respectively with Fig- 
ures 14A - D and represent a one-dimensional Fourier 
transform cut through each of the "scenes", thus clearly 
illustrating the spatial frequencies that have been 

25 restored to the reconstructed image. 

[0053] Figures 1 6A - C represent the application of the 
non-linear method to a thinned aperture system. In this 
case, the thinned aperture configuration is an annulus. 
It should be noted, however, that the method may be uti- 

30 iized with any thinned aperture design. Figure 16A rep- 
resents a computer generated ground scene (i.e. the 
truth scene) . The blurred image of that scene is then 
depicted in Figure 16B, while the final reconstructed, 
super-resolved image is shown in Figure 16C. 

35 [0054] Figures 17A - C represent images of figures 
taken from a CCD camera. Figure 17A represents the 
truth scene (a picture of a toy spacemen), while Figure 
17B shows the blurred image of the scene (observed 
through a small aperture). Figure 17C represents the 

40 reconstructed image, and Figure 1 7D shows the magni- 
tude of the difference of the two-dimensional Fourier 
transform between the truth scene in Figure 17A and 
the blurred image of Figure 17B. Figure 17E shows the 
difference between the truth scene and the first stage of 

45 reconstruction (i.e. the deconvolved figure), while Fig- 
ure 17F shows the magnitude of the difference of the 
two-dimensional Fourier transform and the truth scene 
for the reconstructed, super resolved image. Note that 
black indicates a 0 difference, which is the desired 

t 5o result, while white indicates a maximum difference. As 
one can see from a comparison of Figures 1 7D, E and 
F. the radius of Figure 17D corresponds to the cutoff of 
the optical system or camera, and the deconvolved 
image frequencies in Figure 1 7E have been enhanced 

55 inside the cutoff but remain zero outside the cutoff. The 
super-resolved figure in Figure 17F has further 
improved the image by restoring frequencies outside 
the cutoff as can be shown by the increased blackness 
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of the figure with respect to either Figures 17D or 17E. 
This is a clear demonstration that super resolution has 
occurred. Figures 18A-B represent a graphical illustra- 
tion of the truth, blurred, and reconstructed super- 
resolved images for SNR values of 50 and 100 respec- 5 
tively. Figures 18A and B show that the non-linearly 
reconstructed images closely parallel the truth images. 
[0055] In an alternative embodiment, the reconstruc- 
tion approach using a linear transformed method is now 
described. When reconstructing either the background 10 
scene or the localized object, the imaging system is 
mathematically characterized by a linear operator rep- 
resented by a matrix. To restore either a background 
scene or the localized object, an inverse imaging matrix 
corresponding to the inverse operator must be con- 15 
structed. However, due to the existence of system 
noise, applying the inverse imaging matrix to the image 
is intrinsically unstable and results in a poorly recon- 
structed image. However by applying a constrained 
least squares procedure such as Tikhonov regulariza- 20 
tion, a regularized pseudo-inverse (RPI) matrix may be 
generated. Zero-order Tikhonov regularization is prefer- 
ably used, although higher order Tikhonov regulariza- 
tion may sometimes give better results. The details of 
this are not described here, as Tikhonov regularization 25 
is well-known in the art. 

[0056] A key quantity used to construct the RPI matrix 
is the regularization parameter which controls the image 
restoration. Larger parameter values protect the 
restored image from the corrupting effects of the optical 30 
system but result in a restored image which has lower 
resolution. An optimum or near optimum value for the 
regularization parameter may be derived automatically 
from the image data. Singular value decomposition 
(SVD) of the imaging operator matrix may be used to 35 
compute the RPI matrix. By estimating the noise or 
error level of the degraded image the singular values of 
the matrix determine the extent to which the full informa- 
tion in the original scene may be recovered. Note that 
the use of the SVD process is not essential in determin- 40 
ing the RPI matrix, and other methods such as QR 
decomposition of the imaging matrix may also be used 
to achieve essentially the same result. 
[0057] The imaging matrix size increases approxi- 
mately as the square of the image size, and the compu- 45 
tational burden associated with forming the RPI of the 
imaging matrix quickly becomes intolerable. However, in 
the special case that the image and object fields are the 
same size and are sampled at the same intervals (as is 
the case here), the imaging matrix can be expanded so 
into circulant form by inserting appropriately positioned, 
additional columns. A fundamental theorem of matrix 
algebra is that the Fourier transform diagonalizes a cir- 
culant matrix. This allows the reduction of the image 
reconstruction algorithm to a pair of one dimensional 55 
fast Fourier transforms, followed by a vector - vector 
multiplication, and finally an inverse one dimensional 
fast Fourier transform. This procedure allows the image 



restoration by this .Tikhonov regularization technique to 
be done entirely in the Fourier transform domain, dra- 
matically reducing the time required to compute the 
reconstructed image. Figure 19 provides an illustration 
of the steps taken to obtain the reconstructed image 
using the linear transform method. 
[0058] A flow diagrams of the Unear Algebra Tech- 
nique according to the present invention are shown in 
Figures 19a,b. Referring now to Figure 19a, the tech- 
nique first converts the imaging data collected by the 
optical system of the sensor into a matrix of the form 
g 1 (ij>£ m 2 n h(l-mj-n)f(m,n)+n 1 (1,0 .where h is a 
matrix representation of the point spread function of the 
optical system and f is the matrix representation of the 
unblurred background with the embedded object, while 
is the matrix representation of the additive white 
noise associated with the imaging system (module 12). 
Next, imaging data g 2 comprising the image scene data 
which contains only the background data is obtained in 
theformof g 2 ( j j I3 5: m 2: n h (l-mJ-n)b(m,n)+n 2 (i,j) where 
b is a matrix representation of the unblurred background 
data taken alone and n 2 is a matrix representation of 
additive system white noise (module 14). Both and g 2 
are then low-pass filtered to the cut-off frequency of the 
optical system to reduce the effects of noise (module 
15). Module 16 then shows the subtraction step 
whereby the matrix representation (g 3 ) of the difference 
between blurred scene data containing the background 
and object of interest (g^ and the blurred scene con- 
taining only the background data (g 2 ) is formed as (gr 
g 2 ). Next, the position and size of the object of interest 
is specified by choosing x,y coordinates associated with 
image matrix (Q^-g 2 ) (module 18). A segment of suffi- 
cient size to contain the blurred object in its entirety is 
then extracted from the matrix representation of (grgs). 
as shown in Module 20. That is, an area equal to the 
true extent of the local object plus its diffracted energy is 
determined. An identically located segment (i.e. seg- 
ment having the same x,y coordinates) is extracted from 
the blurred background scene matrix g 2 as shown in 
module 22. The two image segments output from mod- 
ule 15 and 20 are then input to module 24 to restore g 2 
and (g-i -g 2 ) using nth order Tikhonov regularization. The 
restored segments are then added together as shown in 
step 26 and the area containing the restored object of 
interest is extracted therefrom, as shown in module 28. 
[0059] The resulting reconstructed image includes 
much of the spatial resolution which was lost due to dif- 
fraction blurring effects. 

[0060] It should be noted that the present invention is 
not limited to any one type of optical sensor device. The 
principles which have been described herein apply to 
many types of applications which include, but are not 
limited to. Optical Earth Resource Observation Systems 
(both Air and Spaceborne), Optical Weather Sensors 
(both Air and Spaceborne), Terrain Mapping Sensors 
both Air and Spaceborne), Surveillance Sensors (both 
Air and Spaceborne), Optical Phenomenology Systems 
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(both Air and Spaceborne), Imaging Systems that utilize 
optical fibers such as Medical Probes, Commercial 
Optical Systems such as Television Cameras, Tele- 
scopes utilized for astronomy and Optical Systems uti- 
lized for Police and Rescue Work. The imaging 
system/method disclosed herein may be used on a sat- 
ellite, and a means for controlling the satellite which is 
located on the earth may be provided. 
[0061] While the inversion has been particularly 
shown and described with reference to preferred 
embodiments thereof, it will be understood by those 
skilled in the art that changes in form and details may be 
made therein without departing from the spirit and 
scope of the present invention. 

Claims 

1. In an optical system having a detector means and 
processor means in which image data is obtained 
comprising scene data containing an object to be 
reconstructed, and background data of the same 
scene, an improved method for increasing the spa- 
tial resolution of the imaging data produced by the 
optical system, the improvement therewith compris- 
ing: 

obtaining a reconstructed background image, 
and 

low pass filtering the scene data containing the 
object to be reconstructed and using the recon- 
structed background image to eliminate the 
background data from the image data to obtain 
a reconstructed image of the object with 
increased spatial resolution.- 

2. In an optical system having a detector means and 
processor means in which image data is obtained 
comprising scene data containing an object to be 
reconstructed, and background data of the same 
scene, an improved method for increasing the spa- 
tial resolution of the imaging data produced by the 
diffraction limited optical system, the improvement 
therewith comprising: 

filtering the noisy blurred background data (D2) 
of the same scene to obtain noise suppressed 
data: 

applying estimations of point spread functions 
associated with the noise suppressed data and 
optical system to the noise suppressed data to 
obtain a reconstructed background image 
OrM); and 

low pass filtering the noisy blurred scene data 
containing the object to be reconstructed and 
using the reconstructed background image to 
eliminate the background data from the image 
data to obtain a reconstructed image of the 
object with increased spatial resolution. 



3. The method according to claim 2, wherein the step 
of filtering to remove noise from D2 comprises 
using the modified method of sieves to remove said 
noise, using the equations; 

5 

l(x) = X y h 0 (x-y)D2(y) 

D3(x) = S y h 3 (x-y)D2(y) 

io D4(x) = 2 y h 4 (x-y)D2(y) 

where h 3 and h 4 are two and three pixel wide 
point spread functions for removing noise by aver- 
aging adjacent pixels together, and where h 0 repre- 
15 sents the optical system point spread function. 

4. The method according to claim 2, wherein the 
reconstructed background l r (x) is obtained using 
the Richardson-Lucy reconstruction technique. 

20 

5. The method according to claim 3, further compris- 
ing constructing new point spread functions (h 3T , 
h 4T ) to account for combined effect of the method of 
sieves two and three pixel wide point spread func- 

25 tions, and the optical system point spread func- 
tions. 

6. The method according to claim 5. wherein said 
noise suppressed data is used as an estimate of 

30 the true background scene data and wherein the 
step of reconstructing the background I^X) further 
comprises: 

(a) blurring the estimate of the true background 
35 scene data; 

(b) dividing on a pixel by pixel basis the noisy 
scene data (D3, D4) by the blurred estimate of 
the true background scene data to create a 
new array (T3, T4); 

40 (c) correlating said new array with the complete 

method of sieves and optical system point 
spread functions and multiplying the result on a 
pixel by pixel basis with the current estimate of 
the true background scene data to provide a 

45 new estimate of said true background scene 

Z(x) ; and 

(d) taking the new estimate of the true back- 
ground scene to be the reconstructed back- 
ground scene l r (x). 

50 

7. The method according to claim 6, further compris- 
ing the step of determining whether a threshold 
number of iterations has been met and repeating 
the steps of claim 6 until said threshold is achieved. 

55 

8. The method according to claim 6, wherein the noise 
suppressed values D3 and D4 are used as the first 
estimate of the true background scene. 
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9. The method according to claim 6, wherein the step 
of correlating the new array to obtain a new esti- 
mate of the true background scene further com- 
prises using said new point spread functions, h 3T 
and h 4T , where 

Z3(x) = £ Y T3(y)h 3T (y-x) 

Z4(x) = L Y T4(y)h 4T (y-x) 

Z(x) = l n (x)[Z3(x) + Z4(x)]/2 

10. The method according to claim 2, wherein the step 
of eliminating the background data further com- 
prises: 

(a) separating out a first estimate, D 0 (x), of 
reconstructed object data from the filtered D1 
data; 

(b) replacing reconstructed background scene 
pixels by estimate reconstructed object pixels 
at predetermined object positions to obtain an 
image indicative of the reconstructed back- 
ground and estimated reconstructed object 
(S(x)); 

(c) blurring the combination of the recon- 
structed background and estimated recon- 
structed object to obtain a blurred image l s (x); 

(d) dividing on a pixel basis the filtered D1 
scene data by the blurred combination of the 
reconstructed background and estimated 
reconstructed object data to obtain a new array 
of image data N(x); 

(e) correlating said new array N(x) with the opti- 
cal system point spread function and multiply- 
ing for each pixel specified by the current 
estimate of the reconstructed object to provide 
a new estimate of said reconstructed object; 

(f) taking the new estimate of the reconstructed 
object to be the reconstructed image of the 
object having increased spatial resolution. 

1 1 . The method according to claim 1 0, further compris- 
ing the step of determining whether a threshold 
number of iterations has been met and repeating 
steps b-f of claim 10 until said threshold is 
achieved. 

1 2. The method according to claim 10, wherein the step 
of separating out a first estimate of the recon- 
structed object from the fitter data D1 , further com- 
prises using binmap values specifying the position 
in a scene of the object to be resolved in combina- 
tion with the filtered data (D f (x)) to obtain D 0 (X), 
where 

D 0 (x) = binmap (x) • D t (x). 



1 3. The method according to claim 1 2, wherein the pre- 
determined object positions for replacing the recon- 
structed background scene pixels by the estimated 
reconstructed object pixels S(x) are specified by the 

5 binmap values, where 

S(x) - I r (x) (binmap(x)-l ) + binmap (x)D 0 (x). 

1 4. The method according to claim 1 3. wherein the step 
10 of blurring the combination of the reconstructed 

background and estimated reconstructed object 
data uses the optical system point spread function, 
where 

is l s (x) = £ y h 0 (x-y)S(y) 

15. In an optical system having a detector means and 
processor means in which image data is obtained 
comprising noisy blurred scene data containing an 

20 object to be reconstructed, and noisy blurred back- 
ground data of the same scene; a method for 
increasing the spatial resolution of the imaging data 
produced by the diffraction limited optical system, 
said method comprising the steps of: 

25 

converting said imaging data into a matrix (g; j); 
regularizing said matrix by performing nth order 
Tikhonov regularization to said matrix to pro- 
vide a regularized pseudo-inverse (RPI) matrix; 
30 and 

applying said RPI matrix to said matrix to pro- 
vide a reconstructed image of said object. 

16. The method according to claim 15, wherein said 
35 RPI matrix is determined via a singular value 

decomposition (SVD) of the imaging matrix 
obtained from the point spread function. 

17. The method according to claim 15, wherein said 
40 RPI matrix is determined using QR decomposition 

of the imaging matrix. 

18. A method for substantially increasing the spatial 
resolution of imaging data produced by a diffraction 

45 limited optical system said method comprising the 
steps of: 

converting said imaging data into a first matrix 
g 1( comprising background data and object of 
so interest data, and a second matrix ^ compris- 

ing background data; 

subtracting the second matrix g 2 from the f irst 
matrix to obtain a third matrix g 3 indicative of 
the difference between said first and second 
55 matrices, 

specifying a position and size of said object of 

interest in the third matrix g 3 ; 

extracting a segment having coordinates which 
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completely includes said blurred object from 
said third matrix g 3 ; 

performing nth order Tikhonov regularization 
on each said extracted segment and said sec- 
ond matrix to restore said segment, adding 5 
said restored segments, and extracting an area 
of said added restored segment containing 
said object of interest to obtain the restored 
image data containing a restored object of 
interest. 10 

19. The method according to claim 18, wherein said 
first matrix comprises an image scene containing 
an object of interest and background data, and 
wherein said second matrix comprises said image is 
scene containing only said background data. 

20. The method according to claim 18, further compris- 
ing the step of low pass filtering said first and sec- 
ond matrices to the system cutoff frequency. 20 

21 . The method according to claim 1 9, wherein the step 
of extracting a segment having coordinates which 
completely includes said blurred object comprises 
extracting an area equal to the extent of the object 25 
of interest plus the diffracted energy associated 
with said object of interest. 

22. An optical system having a detector means and 
processor means in which image data is obtained 30 
comprising scene data containing an object to be 
reconstructed, and background data of the same 
scene, an improved apparatus for increasing the 
spatial resolution of the imaging data produced by 
the optical system, the improvement therewith com- 35 
prising: 

means for obtaining a reconstructed back- 
ground image, and 

means for low pass filtering the scene data 40 
containing the object to be reconstructed and 
using the reconstructed background image to 
eliminate the background data from the image 
data to obtain a reconstructed image of the 
object with increased spatial resolution. 45 

23. In an optical system having a detector means and 
processor means in which image data is obtained 
comprising noisy blurred scene data containing an 
object to be reconstructed (D1), and noisy blurred so 
background data (D2) of the same scene, an 
improved apparatus for increasing the spatial reso- 
lution of the imaging data produced by the diffrac- 
tion limited optical system, the improvement 
therewith comprising: 55 

means for filtering the noisy blurred back- 
ground data (D2) of the same scene to obtain 



noise suppressed data: 

means for applying estimations of point spread 
functions associated with the noise suppressed 
data and optical system to the noise sup- 
pressed data to obtain a reconstructed back- 
ground image (l r (x)); and 
means for low pass filtering the noisy blurred 
scene data containing the object to be recon- 
structed and using the reconstructed back- 
ground image to eliminate the background data 
from the image data to obtain a reconstructed 
image of the object with increased spatial reso- 
lution. 

24. The apparatus of claim 23, wherein the means for 
filtering to remove noise from D2 comprises means 
for using the modified method of sieves to remove 
said noise, using the equations: 

• l(x)=r y h 0 (x-y)D2(y) 

D3(x) = X y h 3 (x-y)D2(y) 

D4(x) = X y h 4 (x-y)D2(y) 

where h 3 and h 4 are two and three pixel wide 
point spread functions for removing noise by aver- 
aging adjacent pixels together, and where h 0 repre- 
sents the optical system point spread function. 

25. The apparatus of claim 23, wherein the recon- 
structed background l^x) is obtained using a 
means for implementing the Richardson-Lucy 
reconstruction technique. 

26. The apparatus of claim 24, further comprising 
means constructing new point spread functions 
(h 3T , h 4X ) to account for combined effect of the 
method of sieves two and three pixel wide point 
spread functions, and the optical system point 
spread functions. 

27. Ttie apparatus of claim 26, wherein said means to 
obtain the reconstructed background l r (x) further 
comprises: 

(a) means for blurring the estimate of the true 
background scene data; 

(b) means for dividing on a pixel by pixel basis 
the noisy scene data (D3. D4) by the blurred 
estimate of the true background scene data to 
create a new array (T3. T4); 

(c) means for correlating said new array with 
the complete method of sieves and optical sys- 
tem point spread functions and multiplying the 
result on a pixel by pixel basis with the current 
estimate of the true background scene data to 
provide a new estimate of said true background 



11 

BNSDOCID: <EP.„ 0959433A2 I. > 



21 



EP 0 959 433 A2 



22 



scene Z(x); and 

(d) means for taking the new estimate of the 
true background scene to be the reconstructed 
background scene l r (x). 

28. The apparatus of claim 27, further comprising 
means for determining whether a threshold number 
of iterations has been met and repeating the steps 
of claim 27 until said threshold is achieved. 

29. The apparatus of daim 27, wherein the noise sup- 
pressed values D3 and D4 are used as the first esti- 
mate of the true background scene. 

30. The apparatus of claim 27, wherein the means for 
correlating the new array to obtain a new estimate 
of the true background scene further comprises 
means for using said new point spread functions , 
h 3T and h 4T , where 

Z3(x) = 5: y T3(y)h aT (y.x) 
Z4(x) = Z y T4(y)h 4T (y-x) 
Z(x) = l n (x)[Z3(x) + Z4(x)]/2 

31. The apparatus of claim 23. wherein the means for 
eliminating the background data further comprises: 

(a) means for separating out a first estimate, 
D 0 (x), of reconstructed object data from the fil- 
tered D1 data; 

(b) means for replacing reconstructed back- 
ground scene pixels by estimate reconstructed 
object pixels at predetermined object positions 
to obtain an image indicative of the recon- 
structed background and estimated recon- 
structed object (S(x)) ; 

(c) means for blurring the combination of the 
reconstructed background and estimated 
reconstructed object to obtain a blurred image 
IsM; 

(d) means for dividing on a pixel basis the fil- 
tered D1 scene data by the blurred combina- 
tion of the reconstructed background and 
estimated reconstructed object data to obtain a 
new array of image data N(x); 

(e) means for correlating said new array N(x) 
with the optical system point spread function 
and multiplying for each pixel specified by the 
current estimate of the reconstructed object to 
provide a new estimate of said reconstructed 
object; 

(f) means for taking the new estimate of the 
reconstructed object to be the reconstructed 
image of the object having increased spatial 
resolution. 



32. The apparatus of claim 31, further comprising 
means for determining whether a threshold number 
of iterations has been met and repeating steps b-f 
of claim 31 until said threshold is achieved. 

5 

33. The apparatus of claim 32, wherein the means for 
separating out a first estimate of the reconstructed 
object from the filter data D1, further comprises 
means for using binmap values specifying the posi- 

10 tion in a scene of the object to be resolved in com- 
bination with the filtered data (Df(x)) to obtain D 0 (X), 
where 

D 0 (x) = binmap (x) • D f (x). 

75 

34. The apparatus of claim 33, wherein the predeter- 
mined object positions for replacing the recon- 
structed background scene pixels by the estimated 
reconstructed object pixels S(x) are specified by the 

20 binmap values, where 

S(x) = l r (x) (binmap(x)-l) + binmap (x)D 0 (x). 

35. The apparatus of claim 34, wherein the means of 
25 blurring the combination of the reconstructed back- 
ground and estimated reconstructed object data 
uses the optical system point spread function, 
where 

so l s (x) = X y h 0 (x-y)S(y) 

36. In an optical system having a detector means and 
processor means in which image data is obtained 
comprising noisy blurred scene data containing an 

35 object to be reconstructed, and noisy blurred back- 
ground data of the same scene; an apparatus for 
increasing the spatial resolution of the imaging data 
produced by the diffraction limited optical system, 
comprising: 

40 

means for converting said imaging data into a 
matrix (g j( j); 

means for regularizing said matrix by perform- 
ing nth order Tikhonov regularization to said 
45 matrix to provide a regularized pseudo-inverse 

(RPI) matrix; and 

means for applying said RPI matrix to said 
matrix to provide a reconstructed image of said 
object. 

so 

37. The apparatus of claim 36, wherein said RPI matrix 
is determined via a singular value decomposition 
(SVD) of the imaging matrix obtained from the point 
spread function. 

38. The apparatus of claim 36, wherein said RPI matrix 
is determined using QR decomposition of the imag- 
ing 20 matrix. 
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39. An apparatus for substantially increasing the spatial 
resolution of imaging data produced by a diffraction 
limited optical system comprising : 

means for converting said imaging data into a 5 
first matrix g-i , comprising background data and 
object of interest data, and a second matrix g 2 
comprising background data; 
means for subtracting the second matrix g 2 
from the first matrix to obtain a third matrix 10 
g 3 indicative of the difference between said first 
and second matrices, 

means for specifying a position and size of said 
object of interest in the third matrix g 3 ; 
means for extracting a segment having coordi- is 
nates which completely includes said blurred 
object from said third matrix g 3 ; and 
means for performing nth order Tikhonov regu- 
larization on each said extracted segment and 
said second matrix to restore said segment. 20 
adding said restored segments, and extracting 
an area of said added restored segment con- 
taining said object of interest to obtain the 
restored image data containing a restored 
object of interest- 25 

40. The apparatus of claim 39, wherein said first matrix 
comprises an image scene containing an object of 
interest and background data, and wherein said 
second matrix comprises said image scene con- 30 
taining only said background data. 

41. The apparatus of claim 40, further comprising 
means for low pass filtering said first and second 
matrices to the system cutoff frequency. 35 

42. The apparatus of claim 40, wherein the means for 
extracting a segment having coordinates which 
completely includes said blurred object comprises 
means for extracting an area equal to the extent of 40 
the object of interest plus the diffracted energy 
associated with said object of interest. 

43. The method of claims 1 . 2, 6 or 1 5 wherein the opti- 
cal system is on a satellite which is controlled from 45 
the earth. 

44. The apparatus of claims 22, 23, 27 or 36 wherein 
the optical system is on a satellite, further compris- 
ing means, located on the earth, for controlling the 50 
satellite. 
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gi IS THE BLURRED SCENE CONTAINING THE BACKGROUND AND OBJECT OF INTEREST: 
tfi ft J) - £E*> ' - ». - »)/<*.") ♦ »,(*. /) . (/ = U s 1.2....7) 

WHERE h IS THE POINT-SPREAD FUNCTION OF THE OPTICAL SYSTEM 

/ IS THE (UNBLURRED) BACKGROUND WITH EMBEDDED OBJECT 
n t IS ADDITIVE SYSTEM NOISE 



g> IS THE BLURRED SCENE CONTAINING THE BACKGROUND ONLY: 

WHERE b IS THE (UNBLURRED) BACKGROUND ALONB 
n, IS ADDITIVE SYSTEM NOISE 



\ 



FORM (r,-,,) 



SPECIFY POSITION AND SIZE OF THE OBJECT OF INTEREST 



- Jf 



RACT FROM (g t - g j ) A SEGMENT OF SUFFICIENT SIZE TO CONTAIN THE BLURRED OBJECT COMPLETELY 
(i.e., AN AREA EQUAL TO THE TRUE EXTENT OF THE OBJECT PLUS ITS DIFFRACTED ENERGY) 



J" EXTRACT FROM g, AN IDENTICALLY-LOCATED SEGMENT 



RESTORE THE TWO SECBdENTS USING o*-ORDER TKHONOV REGULARIZATION 








1 ADD THE TWO RESTORED SEGMENTS 










| EXTRACT THE AREA CONTAINING THE RESTORED OBJECT OF INTEREST *"* 




/4 
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